Uncertainty management in a decision-making system

ABSTRACT

Systems and methods are provided for constructing and evaluating a story of interest. The system includes a plurality of decision algorithms. Each decision algorithm is operative to quantify at least one category of uncertainty associated with the story of interest as a set of at least one uncertainty parameter. An uncertainty management component is operative to reconcile the sets of uncertainty parameters from the plurality of decision algorithms as to produce a global uncertainty parameter for each of the plurality of uncertainty categories for the story of interest.

TECHNICAL FIELD

The present invention relates to artificial intelligence and, moreparticularly, to systems and methods for managing uncertainty in adecision-making system.

BACKGROUND OF THE INVENTION

In the past, decisions frequently had to be made on minimal amounts ofavailable data. Information traveled slowly, and the scope of theinformation available was within a scale that could be considered by ahuman mind. Frequently, the greatest problem facing a decision-maker wasa paucity of information. Advances in information gathering andtransmittal technologies have reversed this trend, making it easier togather large amounts of information pertaining to a particular problem.A major task facing modern day decision-makers is filtering andorganizing the received information into a useful form. But perhaps themost significant challenge is determining how to process the data,accounting for the inherent uncertainties in the data such as conflicts,false information, ambiguities, errors, measurement biases, etc.

While automated classification and decision-making systems have becomeincreasingly sophisticated, the human mind still outperforms automatedsystems on any real-world tasks that require judgment. One limitation ofhuman decision-making, however, is the inability of human beings toconsider a large number of factors simultaneously. Another limitationexperienced by human decision-makers is their inability to correctlyanalyze information with inherent uncertainties or worse, uncertaintyinduced by processing. Decision-makers often find it difficult tocombine large amounts of evidence mentally, and the human tendency is topostpone risky decisions when data are incomplete, or jump toconclusions and refuse to consider conflicting data. Accordingly,automated methods of organizing, combining, correlating, and displayingdata that account for the uncertainties in the data can greatly aidhuman decision-makers.

In attempting to structure and filter the data presented to a humandecision-maker, an unfortunate tendency of many decision support systemsis to oversimplify the situation presented to the decision-maker. Whileany real-world decision must consider many different types ofuncertainty, this uncertainty is often hidden from the decision-maker byeliminating the context of the information or presenting a singleuncertainty value by thresholding. This leaves the decision-makerwithout explicit information about the uncertainty regarding each “fact”presented as relevant to the pending decision. Implicit information,data thresholding, and loss of context can force the decision-maker toguess about such uncertainty in arriving at a decision or give thedecision maker a false sense of security about the situation and theirdecision. Unfortunately, this can result in sub-optimal decisions,because vital information has in effect been hidden from thedecision-maker by the automation system. A parallel situation pertainswith regard to automated tools that perform analysis of a situation, andmake decisions or recommendations—current practice tends to “hide” thefull range of interpretations of the input data, leading to inferior andeven inaccurate decisions and recommendations.

SUMMARY OF THE INVENTION

In accordance with one aspect of the present invention, an assisteddecision-making system for constructing and evaluating a story ofinterest is provided. The system includes a plurality of decisionalgorithms. Each decision algorithm is operative to quantify at leastone category of uncertainty associated with the story of interest. Anuncertainty category is quantified using at least one uncertaintyparameter. An uncertainty management component is operative to reconcilethe sets of uncertainty parameters from the plurality of decisionalgorithms as to produce a global uncertainty parameter for each of theplurality of uncertainty categories for the story of interest.

In accordance with another aspect of the present invention, a method isprovided for managing uncertainty in an assisted decision-making system.A story of interest is constructed, comprising an structured argumenthaving a plurality of associated elements, links between elements,evidence supporting the plurality of elements, and a plurality ofevidence sources. At least two categories of uncertainty associated withthe story of interest are evaluated to produce respective globaluncertainty parameters. The global uncertainty parameters are displayedto a decision-maker in conjunction with the relevant decisionparameters.

In accordance with yet another aspect of the present invention, acomputer readable medium is provided. The computer readable mediumcontains a plurality of decision algorithms. Each decision algorithm isoperative to quantify at least one category of uncertainty associatedwith the story of interest as a set of at least one uncertaintyparameter. An uncertainty management component is operative to reconcilethe sets of uncertainty parameters from the plurality of decisionalgorithms so as to produce a total uncertainty for each of theplurality of uncertainty categories for the story of interest.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 illustrates an automated system for assisting a humandecision-maker in accordance with an aspect of the present invention.

FIG. 2 illustrates a representation of a belief network, as an exampleof the more general class of structured arguments, in accordance with anaspect of the present invention.

FIG. 3 is a chart illustrating a plurality of uncertainty categories,organized according to their associated properties.

FIG. 4 illustrates an assisted decision-making system, utilizing anexecutable knowledge base, in accordance with an aspect of the presentinvention.

FIG. 5 illustrates an exemplary Kiviat diagram for displayinguncertainty associated with a story of interest to a decision-maker.

FIG. 6 illustrates a methodology for managing uncertainty associatedwith a story of interest within an assisted decision-making system

FIG. 7 illustrates a schematic block diagram of an exemplary operatingenvironment for a system configured in accordance with an aspect of thepresent invention.

DETAILED DESCRIPTION OF INVENTION

The present invention relates to systems and methods for managinguncertainty, such that both the overall uncertainty and a categoricalbreakdown of the uncertainty can be made available to a decision-maker.One or more exemplary implementations of the present invention canutilize a knowledge base that comprises a plurality of stories, where astory is a structured argument augmented by one or more characteristicsof the elements (e.g., states, hypotheses, links, etc.) comprising thestructured argument, the evidence associated with the elements, and thecontent from which the evidence was extracted. The knowledge base can beconstructed from executable elements, such that the stories can bemanipulated and updated in real time with new evidence. For example, aunion or intersection between two stories or portions of stories can beformed, with appropriate parameters calculated from the existing values,without the need for human intervention. This allows the knowledge baseto be meaningfully shared between decision-makers who may utilizedifferent bodies of evidence.

FIG. 1 illustrates an assisted decision-making system 10 in accordancewith an aspect of the present invention. The assisted decision-makingsystem 10 comprises a knowledge base 12, containing stories and portionsof stories provided at least in part by a plurality of decision networks14 and 16. The decision algorithms 14 and 16 can comprise, for example,algorithms for extracting data from various evidence sources, findingrelationships among the extracted data, generating stories from theextracted data, mining the extracted data, and reasoning inferentiallyfrom the stories stored in the knowledge base 12. Evidence, stories, andstory fragments provided by the plurality of decision networks can beutilized by a fusion engine to produce a story of interest for a givenproblem.

The evidence and story fragments provided by the plurality of decisionalgorithms 14 and 16 introduce a degree of uncertainty into the story ofinterest. For a given story fragment (e.g., hypothesis, Markov state,evidence, link, or set of linked hypotheses), this uncertainty can bequantified and categorized at respective uncertainty determinationcomponents 18 and 20 associated with the decision-making algorithms 14and 16. The uncertainty determination components 18 and 20 can beconfigured to account for a plurality of uncertainty categories. Theuncertainty calculated at each uncertainty determination component 18and 20 can be identified as belonging to one of these uncertaintycategories and provided to an uncertainty management component 22. Theuncertainty management component 22 reconciles the uncertainty from theplurality of decision algorithms 14 and 16. For example, the uncertaintymanagement component 22 can aggregate the various calculateduncertainties within the plurality of categories into respective globaluncertainty values, and then produce a unified uncertainty assessment bycombining the global uncertainty values for each of the plurality ofcategories. Each global uncertainty value can be represented as aprobability value and an associated confidence interval, and the globaluncertainty values can be combined via any appropriate means forcombining probability distributions, for example, a root-sum square ofthe belief values and confidence intervals or sensitivity analysis, suchas Monte Carlo. The aggregate uncertainty for each category and thecomposite uncertainty can be provided to a decision-maker to aid in thedecision-making process.

FIG. 2 illustrates a representation of a belief network 30 that can beutilized as part of a story of interest in accordance with an aspect ofthe present invention. It will be appreciated that other types ofstructured arguments, such as such as semantic networks, hidden Markovmodels, and other data fusion algorithms can be utilized to constructstories in accordance with an aspect of the present invention. Thebelief network 30 includes a top layer 32, a first intermediate layer34, a second intermediate layer 36, and a bottom layer 38. The top layer32 includes nodes N1-N6 linked to the first intermediate or hypothesislayer 34 by links or multipliers L1-L10. The first intermediate layer 34includes nodes N7-N10 linked to the second intermediate layer 34 bylinks or multipliers L11-L17. The second intermediate layer 36 includesnodes N11-N13 linked to the bottom layer 38 by links or multipliersL18-L21. Each node represents a given variable and hypothesis associatedwith that variable that can affect the hypotheses of other nodes inlower layers mathematically. Associated with each of the nodes N1-N15are three parameters, which are a belief parameter B, a disbeliefparameter D, and an unknown parameter U. The parameters B, D, and Uconform to the Dempster-Shafer evidential interval such that theparameter B, D and U add up to one for each node N1-N15.

The links represent multipliers or weights of a given parameter on alower node. Link values can be constant, or computed by an algorithm.For example, the belief of node N7 of the first intermediate layer 34depends on the belief of nodes N1, N2, and N3, each multiplied by itsrespective link value L1, L2, and L3. Additionally, the disbelief ofnode N7 of the first intermediate layer 34 depends on the disbelief ofnodes N1, N2, and N3, each multiplied by its respective link value L1,L2, and L3. The unknown is computed based on the Dempster-Shafercombination rule. The belief and disbelief of node N7 then propagate toN11 through link L11, which is combined with the belief and disbelief ofN18 multiplied by link L12 and the belief and disbelief of node N9multiplied by link L14. The belief and disbelief of node N11 thenpropagate to node N14 through link L18 which is combined with the beliefand disbelief of N13 multiplied by link L20. The ignorance, oruncertainty, of each row can be evaluated using the Dempster-Shafercombination rule. Similar propagation occurs to provide the beliefs, thedisbeliefs, and unknowns of the node N15. It will be appreciated that astory, as defined in the present application, can comprise a beliefnetwork that is augmented by information slots, which providecharacteristics of links and nodes that answer a detailed version of thereporter's questions (e.g., who, what, where, when, why, how,)accompanied by the certainty or confidence in each element.

FIG. 3 is a chart 50 illustrating a plurality of uncertainty categories,organized according to their associated properties. The uncertainty 52in a given decision-making system can be divided into two broad sourcecategories, uncertainty introduced from data 54 and uncertaintyintroduced by processes 56 associated with the decision-making system.These source categories can each be broken down into two subcategories.Uncertainty in the data can arise from inaccuracy 58 in the data orincompleteness 60 of the data. Uncertainty in processes can becategorized as conceptual uncertainty 62 and dynamic uncertainty 64. Itwill be appreciated that an understanding of the sources of uncertaintyin a decision-making system can allow a decision-maker to better accountfor the effects of the uncertainty or to take steps to reduce theuncertainty.

Uncertainty due to inaccurate data 58 can be broken down into fouruncertainty categories, falsity 66, obsolescence 68, measurement bias70, and random error 72. Uncertainty due to falsity 66 is uncertaintydue to the possibility that data is factually incorrect. The uncertaintycaused by this possibility can be quantified using historical data for aprovider associated with the data. Uncertainty caused by obsolescence 68stems from the likelihood that the circumstances reflected by the datahave changed since the data was gathered. In an exemplary embodiment,various types of evidence can have associated methods for determiningobsolescence, such as obsolescence curves that determine how thereliability of the evidence decays over time. The uncertainty reflectedby a given item of evidence at any given time can be determinedaccording the age of the evidence (e.g., the length of time since it wasacquired) and these methods. It can be appreciated that the methods aresensitive to the data and the context of that data.

Measurement bias 70 represents possible sampling error in the dataprovided to the decision-making system. Measurement bias 70 cangenerally be qualified based on the sampling size utilized for a givenmeasurement and the variance estimated for the sampled population.Finally, uncertainty associated with random error 72 generally arisesfrom statistical data or data expressed as a possibility. A number ofmethods for quantizing random error 72 in a given probabilitydistribution are known, and one or more of these methods can be utilizedfor quantifying the error in accordance with an aspect of the presentinvention.

Uncertainty due to incomplete data 60 can be categorized as conflict 74,vagueness 76, and information known to be missing 78. Uncertainty due toconflict 74 arises when two items of evidence are contradictory, thatis, when one item of evidence supports a given hypothesis and anothertends to refute the hypothesis. Uncertainty due to conflict 74 can bequantified by assigning belief and disbelief values to the evidence andresolving it via an appropriate combinational rule (e.g., Bayesian orDempster-Shafer). Uncertainty due to vagueness 76 exists when evidenceprovided to the decision-making system is qualified. For example, intext evidence, hedge words, such as possibly or probably, can indicatevagueness. The vagueness 76 can be quantified by assigning moderatingvalues (e.g., fuzzy membership functions) to various qualifiers that canbe expected for a type of evidence. Known missing information 78 isinformation about a given hypothesis that is known to be missing, but isnot practicable to obtain or is not yet recorded. Missing informationcan be quantified in a combination rule as an ignorance (or unknown)value associated with the hypotheses or extrapolated from historicaldata.

Conceptual uncertainty 62 can include three categories, uncertainty dueto mistaken understanding 80, unknown missing information 82 (e.g.,unknown unknowns), and ambiguity 84. The potential for mistakenunderstanding 80 exists when a hypothesis can be interpreted multipleways. This can be quantified by performing a search, for example, ageneral Internet search, to determine if words or phrases in thehypothesis are subject to multiple meanings. The number of multiplemeanings and, possibly, their prevalence in the search, can be utilizedin a formula to produce an uncertainty value. Unknown missinginformation 82 represents information that is missing and is not know tobe relevant to the problem of interest, or a link between hypothesesthat is not known to the decision-maker or the system. These unknownsare difficult to detect, but, in accordance with an aspect of thepresent invention, a general estimate of these unknowns can bedetermined by data mining of other stories in an associated knowledgebase or other raw data sources such as an “open world” source like theInternet to find other, similar stories or data that explain thesituation. Hypotheses and evidence that are present in these stories canbe analyzed to determine analogous evidence and hypotheses that can beincorporated into the story. The amount of new evidence and hypothesesrelative to the amount of existing story elements can be used toquantify the uncertainty attributable to the unknown missing information82. An example of unknown missing information is an unanticipated effectof an action associated with a hypothesis. Uncertainty due to ambiguity84 can exist when evidence cannot be reliably attributed to ahypothesis. For example, in extracting text evidence, ambiguity can beintroduced when the answer to one of the “reporter's questions” (e.g.,who, what, when, where, how) is unclear for a given item of evidence.Uncertainty due to ambiguity 84 can be quantified by reviewing one ormore confidence levels produced by an algorithm used for evidenceextraction.

Uncertainty due to dynamic processes 64 can be divided into twocategories, undecidable uncertainty 86 and chaotic uncertainty 88.Undecidable uncertainty reflects the possibility that the problem isill-posed and that no amount of evidence for the lower rankinghypotheses will provide a threshold level of support for a mainhypothesis. The possibility of undecidable error 86 can be quantified bya Monte Carlo analysis of the existing structure of a story. Adetermined maximum possible belief can be compared to a belief thresholdfor the story to quantify the undecidable error. Chaotic uncertainty 88represents dynamic factors that cannot be predicted but their initialconditions have a significant impact on the outcome, such as weather andsimilar environment factors. Collectively, these factors can be referredto as the “fog-of-war.” The variance on the fog-of-war can be modeled asa probability function, and the effects of the fog-of-war can beestimated through sensitivity analysis (e.g., a Monte Carlo analysis) ofthe story.

FIG. 4 illustrates an assisted decision-making system 100, utilizing anexecutable knowledge base 102, in accordance with an aspect of thepresent invention. The knowledge base 102 is comprised of a plurality ofstories, where each story comprises an structured argument comprising aplurality of elements, at least one link associated with the pluralityof elements, evidence supporting the elements, and a reference (e.g., apointer) to the context from which the evidence was gathered. Each storycan be made executable, such that it can produce mathematicallyconsistent results in response to any change in its associated evidence,belief values, or weights. Accordingly, the stories can be updated andpropagated to multiple decision algorithms in real time, allowing for aflexible exchange between a large number of decision algorithms oranalysts. The illustrated system 100 allows a decision-maker toconstruct and evaluate a story representing a problem of interest to thedecision-maker. In the illustrated example, the story of interest isrepresented as an augmented Dempster-Shafer belief network, but it willbe appreciated that other argument architectures (e.g., Bayesian BeliefNetworks, hidden Markov models, etc.) can be used as well. In accordancewith an aspect of the present invention, the illustrated system 100quantifies a plurality of uncertainty categories that can be associatedwith the story of interest, and allows a decision-maker to account forthe determined uncertainty.

It will be appreciated that the illustrated system 100 can beimplemented as one or more computer programs, executable on one or moregeneral purpose computers such as a computer hard drive, random accessmemory, or a removable disk medium. Accordingly, any structures hereindescribed can be implemented as dedicated hardware circuitry for thedescribed function, as a program code stored as part of acomputer-assessable memory (e.g., magnetic storage media, flash media,CD and DVD media, etc.), or as a combination of both. Functions carriedout by the illustrated system, but not helpful in understanding theclaimed invention, are omitted from this diagram. For example, a systemimplemented as a computer program would require some amount of workingmemory and routines for accessing this memory. Such matters areunderstood by those skilled in the art.

In the illustrated example, evidence and stories can be input into theknowledge base 102 and incorporated into the story of interest 103 in anumber of ways. For example, an information extraction component 104 canbe used to reduce an evidence source, such as a text document or atranscripted conversation, into a desired evidence format. Theinformation extraction component 104 can include hardware scanners,video and audio input jacks, and other hardware devices for receivingthe evidence as well as software operative to recognize and analyze thereceived evidence. This evidence can be incorporated into the story ofinterest 103 or other stories in the knowledge base 102, or new storiescan be assembled in response to the evidence. Stories and portions ofstories can also be provided to the story of interest 103 and theknowledge base 102 using a plurality of inferential algorithms 105-110.For example, the inferencing algorithms 105-110 can include ananalogical reasoning algorithm 109 (e.g., a case-base reasoningalgorithm) that utilizes existing stories in the knowledge base 102 toimprove or repair the story of interest 102. Similarly, an abductivereasoning algorithm 110 (e.g., a data mining algorithm) can determineunknown missing information as additional hypotheses and linkages amongavailable hypotheses in the knowledge base 102 to refine the story ofinterest 103 or other stories within the knowledge base 102.

The information extraction component 104 breaks down raw evidencesources into individual words or phrases, interprets the context andmeaning of the various words or phrases, and uses the extractedinformation to generate a template representing the text segment. Theraw evidence may consist of meta-data, voice input, text, or a number ofother sources. For example, the information extraction component 104 canlook for details relating to an event described in the document, such asthe nature of the event, the cause or motivation for the event, themechanism of the event, the identity of an actor, the location of theevent, the time or date of the event, and the magnitude of the event.Each of these details can be added to a template related to the textsegment. In accordance with one aspect of the invention, the informationextraction component 104 can look for hedge words (e.g., maybe,probably, certainly, never) within the text segment. The informationextraction component 104 can use a co-referencing routine to determinewhat nouns relate to a given hedge word, and use this information todetermine the weight of the evidence associated with the template, inthe form of belief values and disbelief values.

To provide a greatly simplified example, the information extractioncomponent 104 might receive a statement from a bank teller that they arecertain that Mr. Brown has made a deposit of ten-thousand dollars to acorporate account via a personal check at a bank in downtown Atlanta.The information extraction component 104 would locate the nouns withinthe sentence as well as words such as “via,” “at,” and “certain” todetermine the relationships between the various nouns and the locationof certain information. Thus, the question of location can be answeredwith the noun or string of nouns following “at” (e.g., bank in downtownAtlanta). The mechanism of the event can be determined by the nounsfollowing “via” (e.g., personal check). The magnitude of the event canbe determined by finding the numbers (e.g., ten-thousand), and otherdetails can be provided by classifying the remaining nouns (e.g., Mr.Brown is likely the actor; the event is likely a deposit, etc.). Thephrase “almost certain,” once it is verified that it is referring thedeposit, can be used to assign a high belief value to the event.

Several categories of uncertainty associated with a story of interest103 can be recognized and accounted for at this point in thedecision-making process by determining various categories of uncertaintyinherent in the evidence comprising the story of interest. For example,a given evidence source can be vague, containing qualifiers or hedgewords, or ambiguous. Similarly, the evidence could be false, obsolete,contain measurement errors, or a given measurement or sampled populationwithin the evidence source can contain random error. Importantinformation can be missing from a given source, and evidence canconflict within or among evidence sources. It will be appreciated thatknowledge about the amount of uncertainty inherent in the evidence, aswell as the nature of the uncertainty, can be important in thedecision-making process.

In accordance with an aspect of the present invention, the uncertaintydue to each of these factors can be quantified at the informationextraction component 104. Vagueness about the information for a givenitem of evidence can be determined by assigning probability values orfuzzy membership functions to various hedge words associated with theevidence source. Ambiguity can be determined from one or more confidencelevels associated with the information extraction. For example, theambiguity can be determined from a confidence value produced byclassification system used to organize the information into the evidencetemplate or from a tonal evaluation in vocal information. Incompleteevidence, referred to as known missing information, can be identified byevaluating the missing fields within the evidence template. Anuncertainty value for known missing information can be determinedaccording to a-priori′ values for the various missing fields in theevidence template.

The possibility that evidence is false can also be evaluated at theinformation extraction component 104. An associated provider for theevidence can be determined from the raw evidence, and an appropriatepossibility of falsity can be determined from the historical performanceof the provider. Similarly, obsolescence can be determined according toobsolescence curves associated with various types of evidence. Theobsolescence curves dictate a rate at which confidence in a given itemof evidence decays with time. The uncertainty associated withobsolescence can be calculated from the age of the evidence and theobsolescence curves. Measurement errors within the evidence can becalculated from data associated with the evidence, for example, anassociated sampling size or degree of precision in a measurement tool.Generally, these items can be extracted from the evidence itself, forexample, as a confidence interval associated with a given measurement,and converted into uncertainty values. Finally, there can be randomerror associated with a given item of evidence. Like the measurementbias, the data necessary to quantify random error can be extracted fromthe evidence itself, generally as a probability expressed in theevidence.

In accordance with an aspect of the invention, the values determined foreach category of uncertainty associated with a given item of evidencecan be provided to the uncertainty management component 112 forevaluation and reconciled into global uncertainty values for theplurality of uncertainty categories. The uncertainty managementcomponent 112 can operate in concert with a fusion engine 114 associatedwith the system to produce the global uncertainty values. For example,the values associated with a given evidence template for each categorycan be expressed as belief and disbelief values and fused with beliefand disbelief values representing other evidence within the story ofinterest 103 via an appropriate combinational rule to determine a globaluncertainty value for the category within the story.

The extracted evidence can be provided to one or more evidenceclassifiers 116. The evidence classifiers 116 assign the evidence toassociated hypotheses according to the evidence content. It will beappreciated that the evidence classifiers 116 can assign the evidence toone or more existing hypotheses in the knowledge base 102 or generate anew hypothesis (e.g., from a large independent ontology or rule base).The hypotheses within each network can be derived from previouslygenerated networks, new hypotheses added to accommodate additionalevidence, and a priori knowledge of the problem added by an analystthrough a user interface 118. In an exemplary embodiment, the evidenceclassifiers 110 can include a rule-based classifier that classifies theevidence according to a set of user defined rules. For example, rulescan be defined relating to the fields within the template or the sourceof the data. Other classifiers can include, for example, supervised andunsupervised neural network classifiers, semantic network classifiers,statistical classifiers, and other classifier algorithms. Theseclassifiers can be orchestrated to increase the efficiency of theclassification. For example, the rule-based classifier can be appliedfirst, and if a rule is not actuated, the statistical classifier can beused. If a pre-specified probability threshold is not reached at thestatistical classifier, the semantic distance classifier can be appliedand the results shown to the decision-maker for validation. Theextracted data can be stored in the knowledge base or incorporated intothe story of interest at a fusion engine 114. The fusion engine 114fuses the data into the belief network according to one of a pluralityof combinational rules, such as a Dempster-Shafer combinational rule, aBayesian combinational rule, an averaging rule, a logical AND, a logicalOR, an optimistic rule that uses the largest belief value from aplurality of values, and a pessimistic rule that uses the smallestbelief value from a plurality of values.

Conflicting evidence associated with a given hypothesis can beidentified when new evidence and hypotheses are fused into the story ofinterest. For example, a given item of evidence can have associatedbelief or disbelief values relative to a given hypothesis. The beliefand disbelief values associated with conflicting evidence can becombined via an appropriate combination rule associated with the fusionengine to provide overall belief and disbelief values for thehypothesis. The uncertainty due to this conflict can be quantified bycalculating the effect of the hypothesis containing the conflictingevidence on a main hypothesis associated with the story of interestaccording to the link strengths of the intervening links between theconflicted hypothesis and the main hypothesis. Similarly, new hypothesesgenerated at the evidence classifiers can be evaluated to determine thepotential for misunderstanding of the hypotheses as an understandinguncertainty. For example, the Internet or one or more external databasescan be searched for one or more words or phrases associated with a givenhypothesis. The number of meanings determined in the search, thestrength of connection to the hypotheses of interest and theirassociated characteristics, and the prevalence of their usage can beused to quantify the likelihood of misunderstandings of the hypothesisaccording to a predetermined formula or sensitivity analysis method.

The inferencing algorithms 105-110 search the knowledge base 102 forunanticipated story fragments, comprising at least a single newhypothesis, and more typical of story fragments comprising linkedhypotheses and any associated evidence. The inferencing algorithms105-110 can identify unknown missing information within the story ofinterest 103, that is, evidence and hypotheses that are not yetaccounted for within the story of interest, and propose one or morestory fragments from the knowledge base to the decision-maker forpossible incorporation into the story of interest. The plurality ofinferencing algorithms 105-110 can include any of a variety ofappropriate algorithms for evaluating stored data to determinesignificant patterns and trends within the stories comprising theknowledge base 102. In the illustrated example, the plurality ofinferencing algorithms include a link analysis component 105 that can beused to compute link values between existing hypotheses based on thecharacteristics of the hypotheses. When new evidence creates asignificant change in the strength of a link or provides the basis for anew link, the new link data can be provided as candidate for missinginformation that was not previously known.

The plurality of inferencing algorithms can further include anunsupervised clustering algorithm 106. In the unsupervised clusteringalgorithm 106, evidence templates are grouped according to theirassociated characteristics. These clusters can provide an indication ofpreviously unknown hypotheses. Further, the unsupervised clusteringshows the changes in the density of evidence in support of variousexisting hypotheses. This may be an indicator of missing information notpreviously known to be associated with the changed hypotheses. Anotherinferencing algorithm can utilize an evidential reasoner 107 thatreviews new stories in the knowledge base to determine which storyfragments and other bits of evidence that are related to, but notrepresented in, a story of interest 103. The unaccounted for fragmentscan represent unknown missing information.

An abductive reasoner 108 can be used to find the best explanation fornew data using positive and negative examples of evidence drawn frompast stories that relate to the story of interest 103. The output of theabductive reasoner 108 is provided in a hierarchically structured waywith different degrees of granularity computed that compresses theinformation represented in a decision network. New hypotheses or linkedgroups of hypotheses can be extracted from this network as unknownunknowns. One example of an abductive reasoning algorithm is the SUBDUEalgorithm developed at the University of Texas at Arlington. Ananalogical reasoner 109 can examine the similarity between the presentstate of the story of interest 103 and past successful decisionnetworks. The analogical reasoner 109 finds successful cases in theknowledge base that are most similar to the present state of the storyof interest 103 and suggests differences in hypotheses based on thesuccessful cases.

An inductive reasoner 110 computes changes in rules based on newevidence within the knowledge base. The change is based on ruleinduction, using the old evidence supporting the old structure alongwith exceptions to the old rules in the existing evidence to induce newrules that better account for the entire body of old and new evidence.In essence, new rules defining a revised network structure are computedfrom an aggregate of old and new evidence. In one implementation, theWeka algorithm can be utilized within the inductive reasoner 110. Itwill be appreciated that other inferential algorithms can be utilized inaccordance with an aspect of the present invention. The fusion engine114 mathematically reconciles the story of interest 103 in light of thestory fragments provided by the inferencing algorithms 105-110. Theimpact of the unknown missing information on the uncertainty associatedwith the story of interest can be quantified by comparing a new beliefvalue associated with a main hypothesis in the story of interest afterthe new story fragments have been incorporated with the previous beliefvalue for the main hypothesis.

The uncertainty management component 112 can operate in concert with thefusion engine 114 to evaluate the structure of the story of interest 103and quantify an uncertainty due to undecidable uncertainty within thenetwork. In the illustrated example, this is accomplished via a MonteCarlo simulation. Each hypothesis within the story of interest 103 canbe modeled as a probability distribution, according to associatedbelief, disbelief, and uncertainty values. The relationship betweenthese distributions and a probability value associated with a mainhypothesis in the story 103 can be determined via the strength of thelinks connecting the various hypotheses within the stories. The MonteCarlo simulation determines a probability distribution for theproposition that the existing hypothesis within the network will providesufficient support to the main hypothesis as to exceed a thresholdbelief value. This threshold can be determined for a given story by ananalyst through the user interface 118. If the determined likelihood ofachieving the threshold is small, the story 103 is unlikely to providesatisfactory insight into the problem of interest for thedecision-maker. The probability distribution provided by the Monte Carloanalysis provides an indication of the undecidable error within thestory of interest 103.

Chaotic uncertainty for the system can be determined according to asimulation model that accounts for the uncertainty in initial conditions122. The simulation algorithm 122 perturbs link strengths and beliefvalues associated with the story of interest 103 to simulate the effectsof chaotic influences on the decision-making process. A decision-makercan specify the frequency and severity associated with perturbation ofthe conditions according to a desired application at the user interface118. An uncertainty value for chaotic uncertainty can be determined byrepeating the simulation model using different starting conditions overa number of trials in a Monte Carlo analysis. A probability distributionfor the belief value of the main node can be determined from thisanalysis.

Once the various categories of uncertainty for the system have beendetermined, the results for each category and for the total uncertaintyassociated with the story of interest 103 can be displayed to adecision-maker in an appropriate form via the user interface 118. Forexample, the results can be displayed in an appropriately formattedchart, in a graph, or according to one or more changes in the appearanceof the story of interest on a main screen. For example, the hue,brightness, or saturation of one of the nodes can be altered to indicatethe presence of an uncertainty at a given hypothesis that exceeds athreshold value. In an exemplary embodiment, a summary of the totaluncertainty for each category can be displayed to the decision-maker asa Kiviat diagram. It will be appreciated that the uncertainty valuesdetermined for individual hypotheses and items of evidence can beretained during the calculation of the total uncertainty, such that thedecision-maker can review the story 103 to determine any evidence orhypotheses that represent the source of the determined uncertainty whichcan be highlighted and presented to the decision-maker in any number ofways. Accordingly, specific action can be taken to reduce uncertainty inthe decision-making process.

FIG. 5 illustrates an exemplary Kiviat diagram 200 for displayinguncertainty associated with a story of interest to a decision-maker. Thedisplayed uncertainty allows a decision-maker to consider the sources ofthe uncertainty associated with the story of interest and take steps toreduce the uncertainty or mitigate its effects. One method forpresenting the uncertainty to the decision maker, the Kiviat diagram,200 can be divided into four quadrants, each representing a differentclass of uncertainty. For example, a first quadrant 202 can representuncertainty due to inaccurate data, a second quadrant 204 can representuncertainty due to incomplete data, a third quadrant 206 can representuncertainty due to conceptual processing errors, and a fourth quadrant208 can represent uncertainty due to dynamic processing errors. Eachquadrant can contain one or more spokes 210-221 representing specificuncertainty categories within the broad uncertainty category associatedwith the quadrant.

Each of the plurality of spokes 210-221 graphically illustrates anassociated value for an associated uncertainty category. The length of agiven spoke indicates the magnitude of uncertainty associated with anuncertainty category. Accordingly, the magnitude of the uncertainty foreach category can be appreciated relative to the uncertainty for theother categories and a predetermined threshold 230. For example, thelongest spoke 215, representing uncertainty due to obsolete data, canindicate that obsolete data is the largest source of uncertainty in thestory of interest. This may indicate to a decision-maker that a decisionshould be postponed until current data can be obtained for the system.The other spokes 213 and 219 that extend significantly beyond thethreshold represent uncertainty due to known missing data anduncertainty due to unknown missing information. Accordingly, it can beconcluded that additional data would be helpful in reducing theassociated uncertainty of the story of interest, and the cost ofobtaining the desired data can be weighed against the expected reductionin uncertainty.

In view of the foregoing structural and functional features describedabove, methodology in accordance with various aspects of the presentinvention will be better appreciated with reference to FIG. 6. While,for purposes of simplicity of explanation, the methodology of FIG. 6 isshown and described as executing serially, it is to be understood andappreciated that the present invention is not limited by the illustratedorder, as some aspects could, in accordance with the present invention,occur in different orders and/or concurrently with other aspects fromthat shown and described herein. Moreover, not all illustrated featuresmay be required to implement a methodology in accordance with an aspectof the present invention.

FIG. 6 illustrates a methodology 250 for managing uncertainty associatedwith a story of interest within an assisted decision-making system inaccordance with an aspect of the present invention. At 252, a story ofinterest is constructed from existing evidence sources. The story ofinterest can be constructed as an augmented belief network comprising aplurality of hypotheses, at least one link associated with the pluralityof hypotheses, evidence supporting the plurality of hypotheses, and aplurality of sources associated with the evidence. The story of interestcan be constructed, at least in part, by a plurality of decisionalgorithms that construct, evaluate, and organize the plurality ofhypotheses according to the evidence sources and an associated knowledgebase comprising previously constructed stories. It will be appreciatedthat the decision algorithms operate under a certain degree ofuncertainty due to the provided data and the processes used to constructthe story of interest.

At 254, at least two categories of uncertainty associated with the storyof interest are evaluated to produce respective global uncertaintyparameters. It will be appreciated that the uncertainty parameters arenot necessarily scalar values, and they can comprise, for example, aprobability value and an associated confidence interval. For example,uncertainty associated with individual hypotheses can be fused toindicate an overall uncertainty. This uncertainty at each hypothesis cantake a number of forms, such as an understanding uncertaintyrepresenting the likelihood that a hypothesis will be “misunderstood”,when evidence is assigned to the hypotheses. This likelihood can bequantified based on a search of an external database for alternativemeanings of various words and phrases within the hypothesis. Similarly,a Monte Carlo analysis of the story of interest can be used to analyzethe uncertainty due to undecidability, representing the likelihood thatthe story of interest will not produce sufficient support for a mainhypothesis given the present structure of the story.

At 256, the global uncertainty parameters are reconciled to produce acomposite uncertainty parameter for the story of interest. For example,one method for reconciling multiple uncertainty parameters is todetermine the square root of the summed squared values of theuncertainty parameters, and another method would employ a Monte Carloanalysis of probability distributions represented by the uncertaintyparameters. The global uncertainty values and the composite uncertaintyparameter can be displayed to a decision-maker at 258. In an exemplaryimplementation, the uncertainty parameters can be displayed as a Kiviatdiagram.

FIG. 7 illustrates a computer system 300 that can be employed toimplement systems and methods described herein, such as based oncomputer executable instructions running on the computer system. Thecomputer system 300 can be implemented on one or more general purposenetworked computer systems, embedded computer systems, routers,switches, server devices, client devices, various intermediatedevices/nodes and/or stand alone computer systems. Additionally, thecomputer system 300 can be implemented as part of the computer-aidedengineering (CAE) tool running computer executable instructions toperform a method as described herein.

The computer system 300 includes a processor 302 and a system memory304. A system bus 306 couples various system components, including thesystem memory 304 to the processor 302. Multi-processor architecturescan also be utilized as the processor 302 as the process is asynchronousand inherently parallel. The system bus 306 can be implemented as any ofseveral types of bus structures, including a memory bus or memorycontroller, a peripheral bus, and a local bus using any of a variety ofbus architectures. The system memory 304 includes read only memory (ROM)308 and random access memory (RAM) 310. A basic input/output system(BIOS) 312 can reside in the ROM 308, generally containing the basicroutines that help to transfer information between elements within thecomputer system 300, such as a reset or power-up.

The computer system 300 can include a hard disk drive 314, a magneticdisk drive 316, e.g., to read from or write to a removable disk 318, andan optical disk drive 320, e.g., for reading a CD-ROM or DVD disk 322 orto read from or write to other optical media. The hard disk drive 314,magnetic disk drive 316, and optical disk drive 320 are connected to thesystem bus 306 by a hard disk drive interface 324, a magnetic disk driveinterface 326, and an optical drive interface 334, respectively. Thedrives and their associated computer-readable media provide nonvolatilestorage of data, data structures, and computer-executable instructionsfor the computer system 300. Although the description ofcomputer-readable media above refers to a hard disk, a removablemagnetic disk and a CD, other types of media which are readable by acomputer, may also be used. For example, computer executableinstructions for implementing systems and methods described herein mayalso be stored in magnetic cassettes, flash memory cards, digital videodisks and the like. The computer system can include any number of inputdevices for accessing evidence to be processed including audio processorchannels, image recognition and analysis machinery, scanners, and ornetwork connections to information sources such as the Internet or otherdata repositories.

A number of program modules may also be stored in one or more of thedrives as well as in the RAM 310, including an operating system 330, oneor more application programs 332, other program modules 334, and programdata 336.

A user may enter commands and information into the computer system 300through user input device 340, such as a keyboard or a pointing device(e.g., a mouse). Other input devices may include a microphone, ajoystick, a game pad, a scanner, a touch screen, or the like. These andother input devices are often connected to the processor 302 through acorresponding interface or bus 342 that is coupled to the system bus306. Such input devices can alternatively be connected to the system bus306 by other interfaces, such as a parallel port, a serial port or auniversal serial bus (USB). One or more output device(s) 344, such as avisual display device or printer, can also be connected to the systembus 306 via an interface or adapter 346.

The computer system 300 may operate in a networked environment usinglogical connections 348 to one or more remote computers 350. The remotecomputer 348 may be a workstation, a computer system, a router, a peerdevice or other common network node, and typically includes many or allof the elements described relative to the computer system 300. Thelogical connections 348 can include a local area network (LAN) and awide area network (WAN).

When used in a LAN networking environment, the computer system 300 canbe connected to a local network through a network interface 352. Whenused in a WAN networking environment, the computer system 300 caninclude a modem (not shown), or can be connected to a communicationsserver via a LAN. In a networked environment, application programs 332and program data 336 depicted relative to the computer system 300, orportions thereof, may be stored in memory 354 of the remote computer350.

What has been described above includes exemplary implementations of thepresent invention. It is, of course, not possible to describe everyconceivable combination of components or methodologies for purposes ofdescribing the present invention, but one of ordinary skill in the artwill recognize that many further combinations and permutations of thepresent invention are possible. Accordingly, the present invention isintended to embrace all such alterations, modifications, and variationsthat fall within the spirit and scope of the appended claims.

1. An assisted decision-making system for constructing and evaluating astory of interest comprising: a plurality of decision algorithms, eachdecision algorithm being operative to quantify at least one category ofuncertainty associated with the story of interest as at least oneuncertainty parameter; and an uncertainty management component operativeto reconcile the uncertainty parameters from the plurality of decisionalgorithms as to produce a global uncertainty parameter for each of theplurality of uncertainty categories for the story of interest.
 2. Thesystem of claim 1, the plurality of decision algorithms comprising aninformation extraction component that reduces evidence sources into adesired format.
 3. The system of claim 2, the at least one category ofuncertainty quantified by the information extraction component includingan uncertainty due to vague evidence, the information extractioncomponent quantifying the uncertainty according to hedge words foundwithin an associated evidence source.
 4. The system of claim 2, the atleast one category of uncertainty quantified by the informationextraction component including an uncertainty due to evidence andhypotheses that are known to be missing, the information extractioncomponent quantifying the uncertainty according to missing entries in anevidence template associated with the desired format.
 5. The system ofclaim 2, the at least one category of uncertainty quantified by theinformation extraction component including an uncertainty due toambiguous evidence, the information extraction component utilizing aclassification system to reduce the evidence sources and quantifying theuncertainty according to a confidence value generated by theclassification system.
 6. The system of claim 2, the at least onecategory of uncertainty quantified by the information extractioncomponent including uncertainty due to false evidence, the informationextraction component quantifying the uncertainty according to at leastone of the historical performance of a provider of the evidence or acomparison of multiple sources to resolve and discount false evidence.7. The system of claim 2, the at least one category of uncertaintyquantified by the information extraction component including anuncertainty due to obsolete evidence, the information extractioncomponent quantifying the uncertainty according to an associated age ofthe evidence and at least one obsolescence method that defines a rate atwhich confidence in a given item of evidence decays over time.
 8. Thesystem of claim 2, the at least one category of uncertainty quantifiedby the information extraction component including uncertainty due tomeasurement error.
 9. The system of claim 2, the at least one categoryof uncertainty quantified by the information extraction componentincluding uncertainty due to random error.
 10. The system of claim 1,further comprising a knowledge base comprising a plurality of executablestories, a given executable story comprising a belief network containinga plurality of associated hypothesis and evidence associated with the atleast one hypothesis.
 11. The system of claim 10, the system furthercomprising at least one inferencing algorithm that examines existinghypothesis within executable stories stored in the knowledge base todetermine one or more appropriate hypotheses for a body of evidence. 12.The system of claim 11, the at least one category of uncertaintyquantified by the at least one inferencing algorithm including anuncertainty due to unknown missing information.
 13. A method formanaging uncertainty in an assisted decision-making system: constructinga story of interest, the story of interest comprising an structuredargument having a plurality of associated elements, at least one linkassociated with the plurality of elements, evidence supporting theplurality of hypotheses, and a plurality of evidence sources; evaluatingat least two categories of uncertainty associated with the story ofinterest to produce respective global uncertainty parameters; anddisplaying the global uncertainty parameters to a decision-maker. 14.The method of claim 13, further comprising reconciling the globaluncertainty parameters as to produce a composite uncertainty parameterfor the story of interest.
 15. The method of claim 13, furthercomprising: conducting a sensitivity analysis of the story of interestto evaluate the likelihood that the story can produce a satisfactorylevel of belief associated with a main hypothesis associated with thestory of interest; and producing a global uncertainty parameterrepresenting the likelihood that insufficient evidence exists to supportthe main hypothesis associated with the story of interest fromsensitivity analysis.
 16. The method of claim 13, further comprising:searching an external database for at least one phrase associated with agiven element associated with the story of interest; determining anuncertainty parameter that represents an uncertainty due tomisunderstandings of the element based on the results of the search; andreconciling uncertainty parameters representing a plurality ofhypotheses to provide a global uncertainty parameter representinguncertainty due to misunderstood elements.
 17. The method of claim 13,wherein displaying the global uncertainty parameters to thedecision-maker includes displaying uncertainty to the decision-maker viaa graphical representation.
 18. A computer readable medium comprising: aplurality of decision algorithms, each decision algorithm beingoperative to quantify at least one category of uncertainty associatedwith the story of interest as at least one uncertainty parameter; and anuncertainty management component operative to reconcile the uncertaintyparameters from the plurality of decision algorithms so as to produce atotal uncertainty for each of the plurality of uncertainty categoriesfor the story of interest.
 19. The computer readable medium of claim 18,comprising a fusion engine operative to mathematically reconcile thestory of interest when data within the story of interest is altered, thefusion engine being operative to determine an uncertainty parameterrepresenting conflicting evidence for at least one hypothesis comprisingthe story of interest.
 20. The computer readable medium of claim 18,further comprising a simulation component that perturbs at least onevalue within the story of interest according to a model, the simulationcomponent being operative to determine an uncertainty parameterrepresenting chaotic uncertainty by conducting a sensitivity analysis ofthe effects of the perturbation of the at least one value on a mainhypothesis within the story of interest.